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ABSTRACT 

This study analyzed the special education program 
evaluation practices of 49 states and the District of Columbia based 
on states* required reporting under Part B of the Individuals with 
Disabilities Education Act and on state plans. Comparison of the 
results with those from a 1984 study suggests the following trends: 

(1) an increase in the number of states that discuss compliance 
monitoring in the annual evaluation sections of the State Plan and 
Part B Performance Report; (2) an increase in the number of states 
that describe a two-tiered approach to the annual evaluation 
requirement, whereby the local education agencies (LEAs) have 
specific evaluation requirements and the state has specific 
monitoring requirements; (3) an increase in the number of states that 
encourage LEA *'sel f-s tudy** as a pre-monitoring tool or evaluation 
tool; and (4) an increase in the number of states that provide 
technical assistance to LEAs to help them evaluate their programs and 
procedures. Fourteen states are conducting statewide evaluations of 
special education. Some states are investigating ways of collecting 
data on student performance outcomes. Ninety-four percent of states 
continue to use compliance monitoring procedures as a program 
evaluation tool. Three recommendations are offered: a renewed 
emphas i s on program eva luat ion for determining special educat i on 
effectiveness; technical assistance in program evaluation at state 
and local levels; and avoidance of legal restrictions on the types of 
program evaluations that can be c« nducted. 
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STATE PROCEDURES FOR THE EVALUATION OF SPECIAL 
EDUCATION PROGRAM EFFECTIVENESS 



Executive Summary 

The purpose of this study was to begin to examine the efficacy of program evaluation 
under the IDEA by describing the methods currently used by States to conduct their annual 
evaluation of program effectiveness. Information on the evaluation practices of 49 States and 
the District of Columbia was collected from the annual evaluation sections of States’ IDEA Part 
B Performance Reports for 1989-90 and State Plans for the implementation of the IDEA covering 
the 1992 school year. A comparison of the results from this study to those from a 1984 
NASDSE study on this topic suggests the following trends: 

* An increase in the number of States that discuss compliance monitoring in the 
annual evaluation sections of the State Plan and Part B Performance Report. 

• An increase in the number of States that describe a two-tiered approach to the 
annual evaluation requirement, whereby the LEAs have specific evaluation 
requirements and the State has specific monitoring requirements. 

• An increase in the number of States that encourage LEA "self-study". This 
practice was described by different States as a pre-monitoring tool or an 
evaluation tool. 

* An increase in the number of States that provide technical assistance to LEAs to 
help them evaluate their programs and procedures. 

Documents reviewed for this investigation indicate that 16 States are supporting short-term 
studies, sometimes using IDEA-B discretionary monies, to evaluate specific issues, projects, 
and/or practices in special education An additional 14 States have begun to conduct or fund 
statewide evaluations of special education, including studies that focus on the post-school status 
of graduates and on the effectiveness of transition services. 

The Individualized Education Program (lEP) is receiving evaluative attention in all States. 
In order to evaluate the lEP, States select among three general strategies: (a) evaluation based 
on the aimual review process, (b) evaluation by the LEA along with compliance monitoring by 
the SEA (i.e., the two-tiered approach), and (c) evaluation by the SEA, usually collecting data 
firom a sample of lEPs. 

Some States have begun to investigate ways of collecting data on student performance 
outcomes for the purpose of evaluating the effectiveness of special education. Four States, 
Indiana, Michigan, Oklahoma, South Carolina, specifically mention producing comprehensive 
technical assistance or guidance documents that encourage LEAs to use student outcomes in their 
evaluation activities. Three States, Vermont, Utah, and West Virginia, report a process in place 



for collecting statewide information on student outcomes for evaluation purposes. Four more 
States, Connecticut, Delaware, Kentucky, and Maryland, indicate they are in various stages of 
implementing such a process. 

Perhaps the finding of broadest significance for using evaluation to improve special 
education programs and practices is that the majority of States (94 percent) continue to use 
compliance monitoring procedures as a program evaluation tool. In some States, there are clear 
implications that the findings from compliance monitoring- are used as evidence of the 
effectiveness of special education programs and services, and that corrective action following 
compliance monitoring is equated with implementing program improvements. Thus, the 
documents reviewed suggest that program evaluation and compliance monitoring firequently are 
viewed as the same activity. 

There appear to be several reasons why States have not clearly differentiated between 
evaluation and compliance monitoring activities. First of all, the Federal requirements to 
"evaluate the effectiveness" of special education programs, as expressed in both statute and 
regulation, provide no elaboration on how this type of evaluation process is to be conducted, 
what data is to be collected, or how it is to be analyzed or reported. Second, it is 
administratively efficient for many States to use, for as many purposes as possible, the 
voluminous amounts of data that must be collected during compliance monitoring and fiscal 
audits, the child count, and other Federal data reporting requirements. A third possible factor 
influencing the apparent emphasis on "monitoring as evaluation" is that expertise in program 
evaluation may not be readily available in many SEAs or LEAs. At the State level, it clearly 
takes more than just knowledge of evaluation methods; it takes the equipment, personnel, and 
statistical expertise to do meta-analyses of data on multiple variables supplied from multiple 
sources. 

The findings from this study on the status of States’ annual evaluation practices, when 
combined with the reasoning expressed above, suggest several recommendations for future policy 
and practice. 

• There should be a renewed emphasis on program evaluation as a necessary 
tool for determining the effectiveness of speciai education programs and 
services and making informed decisions concerning program improvement 
This is not to say that compliance monitoring should be discontinued. However, 
consideration might be given to promoting a balance between the human and 
fiscal resources necessary for conducting compliance monitoring and program 
evaluation at all levels of special education administration based upon their 
relative contribution to improving outcomes for children and youth with 
disabilities. 

• Technical assistance in the area of program evaluation is needed at the SEA 
and LEL\ levels. Expanding the focus of monitoring activities to include program 
evaluation, as the first recommendation suggests, will not necessarily change the 
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amount of effort required for data collection. It has the potential, however, to 
make the data much more useful as long as there is a certain level of expertise 
available to interpret, report, and make the improvements the data suggest. 

State and Federal laws and regulation should be phrased so as to avoid 
restricting the types of program evaluations that can be conducted by SEAs 
and LEIAs. As the implementation of the IDEA continues to evolve in the 
coming years, the questions of critical importance to improving program quality, 
as well as determining program effectiveness, will need to reflect changes in many 
aspects of the educational system as well as national standards and community 
values. 



STATE PROCEDURES FOR THE EVALUATION OF SPECIAL 
EDUCATION PROGRAM EFFECTIVENESS 



Introduction 



Two pieces of legislation form the Federal mandate for evaluating special education 
programs and services that States provide under Part B of the Individuals ^vith Disabilities 
Education Act (IDEA). The IDEA requires each State Plan to "provide for procedures for 
evaluation at least annually of the effectiveness of programs in meeting the educational needs of 
children with disabilities (including evaluation of individualized education programs)... (Section 
1413(a)(ll)). In addition, Section 435(b)(4) of the General Education Provisions Act (GEPA) 
calls for the State to evaluate the effectiveness of covered programs (including the IDEA) at least 
every three years. 

While there are Federal requirements for evaluation, and often State requirements as well, 
there are many SEAs and LEAs that evaluate special education programs and services expressly 
for the benefits this type of information can provide. To illustrate, Wentling (1989) su^ests five 
purposes of evaluation that have applicability to the administration of special education. First, 
evaluation results can be used to convince or gain support. Evaluation results can make a 
significant contribution to a rationale for allocating resources, revising the regulatory structure, 
or establishing policy in new areas. Decisionmakers can be swayed by convincing evidence, and 
evaluation provides a mechanism for obtaining data to support a position, pro or con. 

Second, evaluation results can be used to educate or promote understanding. There are 
many people who are unaware of or who do not understand the goals, procedures, and successes 
of special education. The need to "promote" special education through evaluation of its outcomes 
is particularly acute in times of leveling or declining fiscal resources occurring in the midst 
educational restructuring. A third potential benefit of program evaluation is securing the 
involvement of key individuals. Sharing the results of an evaluation is an excellent way to spark 
the interest and obtain the commitment of people (e.g., boards of education, parents, other agency 
personnel) who can assist in implementing the recommendations. 

Wentling’s fourth and fifth uses for evaluation results are perhaps the ones most 
commonly associated with evaluation - to improve programs and to demonstrate accountability. 
Both concepts have received a great deal of attention in the past decade. Concomitantly, there 
has been increasing pressure from politicians, professionals, and the general public to define and 
promote education outcomes for all our nation’s school children'. The 1990s has begun as a 



' In this context, the term, "outcomes", refers to the knowledge, techniques, values, and affects that are the 
products of education (Schleisman & King, 1990). 
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decade of reform and restructuring for both regular and special education. Improving student 
performance outcomes has become the focus for many of these efforts. In light of the potential 
for change in the administration and delivery of special education in the upcoming years, it is 
appropriate that State administrators re-examine the status of their program evaluation procedures 
- Do evaluation results help to direct the future of special education? Are the standards for 
program success or failure to some extent based on student outcomes? Can evaluation methods 
be improved? 

The purpose of this study is to begin to examine the efficacy of program evaluation under 
the IDEA by describing the methods currently used by States to conduct their annual evaluation 
of program effectiveness. This report provides information on current and developing practice 
in evaluation, including the use of student outcome measures, that may be shared among States 
and the Regional Resource Centers (RRQ. Specific recommendations for changes in evaluation 
requirements are described in the concluding section of this report. The results of this study can 
facilitate OSEP’s efforts to provide information and technical assistance for program 
improvement under Section 618 of the IDEA and further serve as a "baseline" against which 
future improvements in States’ annual evaluation practices may be measured. 



Method 



Source Documents 

Information on the evaluation practices of 49 States and the District of Columbia comprise 
the data for this study. The primary data sources were States’ IDEA Part B Performance Reports 
for 1989-90 and State Plans for the implementation of the IDEA covering the 1992 school year. 
The Performance Reports were unavailable for four States; however, it was felt that the Ztate 
Plans provided enough information to include these States in the study. Neither source document 
was available for one State and it was omitted from the study. 



Data Collection 

Using the annual evaluation sections of the source documents, data collection proceeded 
with two rounds of content analyses. In the fiist round ’ '*ion was collected from each 
State’s documents regarding (a) the program components being evaluated, (b) the type of data 
collected, (c) the data collection and data analysis procedures employed, (e) the use of the data 
for program improvement, and (f) the use of student outcome measures. The information 
pertinent to each of these variables (when available) was copied directly from the source 
documents using the same or abbreviated wording. The annual evaluation sections of both the 
Performance Report and the State Plan were then attached to the form completed for each State. 
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A second review of the source documents was conducted in order to categorize certain 
aspects of States’ evaluation and/or monitoring practices. This round of data collection recorded 
the presence or absence of certain procedural components, rather than describing processes. 
Table 1 provides a list of the questions used in this analysis and how the data were recorded. 
These particular questions were selected in order to compare similar data collected by NASDSE 
in April of 1984 for publication in a monograph entitled, State Responses to the "Annual 
Evaluation" Requirements for State Plans. As part of Question 5, a specific data source category 
(e.g., l.EA application, site visit report) was marked when it was referred to in the "Annual 
Evaluation" section of a State’s Performance Report or Plan as a data collection instrument for 
either monitoring, program evaluation, or both. The category, complaint reports, was added to 
the present study because of its prevalence in current State systems of evaluation and/or 
monitoring. 



Table 1 



Categories Used in the Secondary Analysis 



1. At what levels were significant monitoring activities conducted? - SEA, 

LEA, or Both 

2. At what levels were significant evaluation activities conducted? - SEA, LEA, 
or F oth 

3. Was there an LEA self-study component? - yes/no 

4. Was the SEA offering technical assistance to the LEA in evaluation? - 
yes/no 

5. Which of the following sources provided evaluation/monitoring data? 

a. LEA applications 

b. Site visit reports by SEA 

c. Data reports (e.g., child count) 

d. Fiscal audits 

e. Complaint reports 
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Results 



This section of the report describes the methods currently used by the States to conduct 
their annual evaluation of program effectiveness. Differences between these data and those 
reported in 1984 in an earlier NASDSE study are discussed. Examples of independent and 
statewide evaluation studies funded by States are provided as part of this description. Completing 
this section is a summary of the results with respect to evaluation of the lEP and outcomes-based 
evaluation. 

For the majority of States in this study, the results to follow represent a combination of 
the evaluation activities reported as accomplished during the 1989-90 school year and those 
proposed in the State Plan for the 1992 school year. The amount of detail provided in either the 
Performance Report or the State Plan with respect to the annual evaluation varied greatly from 
State-to-State. Therefore, it is possible that some States conducted or planned to conduct more 
activities in the area of evaluation than they chose to report. In addition, while there was a 
conscious effort during data collection to discriminate between compliance monitoring and 
program evaluation activities, it was clear that States often used the terms interchangeably. Both 
the lack of detail in the source documents and the lack of precision in the use of terms affect the 
accuracy of these results. 



Monitoring as Evaluation - Then and Now 

The limitations cited above notwithstanding, comparisons with the earlier NASDSE 
study^ show some general trends in State program evaluation activities over the past six to eight 
years. For instance, in the 1984 study, most States (approximately 83 percent) chose to discuss 
compliance monitoring in the "Annual Evaluation" section of their Performance Reports or Plans. 
Similarly, in the current study, 92 percent (46) of the States mentioned compliance monitoring 
in this section. Four States, Utah, Minnesota, West Virginia, and Virginia’ restricted their 
descriptions to program evaluation activities in the "Annual Evaluation" section of their Plans. 
Eight other States (i.e.. New Hampshire, Vermont, Delaware, Maryland, Kentucky’, Kansas, 
Nebraska, California) described significant evaluation efforts beyond monitoring. Put another 
way, although compliance monitoring information was mentioned as a contributor to the annual 
program evaluation conducted by these eight States, it did not appear to be the primary source 
of evaluation data. 



’ The 1984 NASDSE study conUincd an analysis and compendium of written descriptions of the annual 
evaluations planned by 47 States as submitted to OSEP in their FY 1984-86 Plans. 

’ The evaluation system described in the source documents was in the planning and design stage. 
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From the data in this study it appears that the roles assigned to the LEA and SEA in 
monitoring and evaluation also have shifted somewhat since 1984. In the first study NASDSE 
reported that approximately 11 States (one quarter of the sample) described a two-tiered 
arrangement with the LEAs, whereby the LEAs had specific evaluation requirements and the 
State had specific monitoring requirements. In the current study 25 States, or half the sample, 
were reporting this type of system. Montana’s 1990-1992 State Plan (p. 94) provides a concrete 
example of a two-tiered system. It contains the following statement: 

Two methods of evaluating local education agency programs are: 

(1) Local education agencies submitting annual updates of 
program narratives, and 

(2) State education agency on-site visits of local education 
agency programs. 

Accompanying this statement are citations from the Administrative Rules of Montana that 
describe the contents of the LEA program narrative and requirements for the evaluation to be 
conducted at the district level (e.g., type of data, evaluation design, reporting requirements). 
Later passages describe the on-site monitoring activities of the SEA and their role in "evaluating 
the general effectiveness of special education programs and services throughout the state." 

Expanding upon the two-tiered evaluation system, a slightly more complex configuration 
was defined by three States in 1984 and by nine States in the present study. In this system the 
LEAs were delegated evaluation responsibilities and the SEAs had both evaluation and 
monitoring ftinctions. 

Another practice found in more States now than in the past is LEA "self-study". The 
1984 NASDSE report mentioned four States engaging in this practice. In contrast, 18 States 
promote its use in the 1990s. From the present study, it is clear that the procedure has different 
purposes among the States. In several States, the LEA self-study is a pre-monitoring activity to 
screen for problems prior to a site visit from SEA staff. In other States, it is apparent that the 
role of LEA self-study goes beyond compliance monitoring and does serve a program evaluation 
function. For still other States, the distinction between the evaluation and monitoring functions 
of LEA self-study activities are not clear froni the source documents. Many SEAs did and still 
do provide technical assistance to LEAs to help them evaluate their programs and procedures. 
In fact, the percentage of States that mention providing this assistance has grown from 20 percent 
(as reported in 1984) to 38 percent in the current study. 

Because program evaluation was and continues to be equated with compliance monitoring 
in many States, it is not surprising that the sources of SEA "evaluation" data are frequently 
elements of their compliance monitoring systems. For instance, the prior NASDSE study 
reported that most States used LEA applications/plans and on-site visitation in their evaluation 
process. Exact figures from the present study are provided in Table 2. 
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Table 2 



The Number of States Using Monitoring Elements 
as Sources of Evaluation Data 



Monitoring Elements Number of States 



LEA Plans/Applications 


34 


LEA Annual Reports 


10 


Site Visits 


42 


Statewide Data Systems 
(e.g.. Child Count) 


21 


Fiscal Audits 


24 


Complaint Reports 


11 



Monitoring data sources such as LEA applications, site visits, or fiscal audits provide very 
limited information upon which to evaluate the effectiveness of special education programs and 
services. Even in States where there were established guidelines for evaluation by the LEAs, it 
was often unclear as to whether the SEA performed any type of "meta-analysis", although a few 
States referred to "aggregating the data." In other instances, the LEA Annual Performance Report 
(including the evaluation data) related to documenting compliance rather than program 
effectiveness. While reliance on compliance documents for the annual evaluation data appears 
to be the norm, some States are expanding their conceptualization of what constitutes evaluation 
of effectiveness in special education. 



Independent Evaluation Studies Funded by States 

One way in which SEAs conduct evaluation involves funding short-term studies, 
sometimes using IDEA-B discretionary monies, to evaluate specific issues and/or practices in 
special education. Sixteen States described supporting studies of this type. For example, Kansas 
funded a study that examined the effects of classwide peer tutoring on at-risk students’ 
subsequent placement in special education. In Minnesota, a project was funded to evaluate the 
relative effectiveness of direct language therapy and therapy provided through mainstream 
collaboration. Some States employing this evaluation format reported that the results were used 
to influence policy decisions, SEA technical assistance priorities, and/or CSPD plans. Thus, iu 
these instances. States have chosen to conduct evaluations that will guide decisionmaking. 
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The "Annual Evaluation" sections of source documents from 10 States maintained that 
evaluation reports were required from projects funded by IDEA-B discretionary monies, including 
model projects initiated by the LEAs. While there was little direct discussion of how project 
evaluations of this type contributed to the overall evaluation of the effectiveness of special 
education, general statements from some States suggested that evaluations from individual 
projects had the potential to influence policy and engender support for project replication and/or 
continuation. 

Statewide Evaluation Efforts Conducted by the SEA 

The evaluation activities described above usually targeted a specific project, curriculum, 
or issue that may or may not have implications statewide. In approximately 14 SEAs, however, 
efforts to conduct or fund statewide evaluation of special education procedures have begun. In 
Iowa, for instance, a five-year statewide follow-up study of special education graduates has been 
supported by the SEA using Part B discretionary funds. Also, Maryland is supporting a statewide 
evaluation study of the status of transition services in schools and community colleges. A 
burgeoning interest in evaluation at the SEA level has been demonstrated in Minnesota and 
Illinois through the establishment of evaluation units within the departments of education. 

Although it was clear from the materials reviewed for this study that some SEAs actively 
were engaged in program evaluation activities, very little detail was provided in the source 
documents as to the evaluation procedures SEAs were using or proposing to use. Descriptions 
particularly were lacking for data collection procedures, data analyses, and criteria for judging 
effectiveness^. One possible reason for the limited information on the procedures used in 
program evaluation was that in some States the topic(s) under study at the SEA level were 
selected on an annual basis. This practice would preclude describing the topics and specific 
evaluation methods in the State Plan. In Missouri, for instance, the topic for evaluation was 
selected annually based upon the results of SEA monitoring activities, the needs assessment data 
from CSPD, and other sources. Similarly, Minnesota conducted formal studies on "critical" 
issues as needed, and in West Virginia, a small number of evaluation questions were selected 
annually for the purpose of determining the State’s progress in meeting certain statutory 
objectives. 

In another example, the California Education Code calls for annual evaluation studies of 
issues of statewide concern in special education. These studies may include, but are not limited 
to, (1) Pupil performance, (2) Placement of pupils in least restrictive environments, (3) Degree 



* Considerably more infomution on procedures could be gleaned from Stete documents where compliance 
monitoring was equated with evaluation. For these States, it was relatively clear what components were "evaluated", 
and what and how the daU were collected (refer to Table 2). There was, however, little information on how these 
daU were analyzed. With respect to interpreting the data, it appeared that in some cases being "in compliance" was 
proof of "effectiveness". 
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to which services identified in individualized education programs are provided, (4) Parent, pupil, 
teacher, program specialist, resource specialist, and administrator attitudes toward services and 
processes provided, and (5) Program costs. 



Evaluating the Effectiveness of the lEP 

While in California and other States the program components to be evaluated are 
determined annually, one component of special education receiving some evaluative attention in 
all States is the lEP which receives specihc mention in the IDEA and supporting regulation. All 
States made some reference to "evaluating the effectiveness of the lEP" in their State Plans. 
Descriptions of how this was accomplished varied from none to several pages of procedures, 
questions, and reporting requirements; however, there seemed to be three general approaches to 
evaluating the lEP that were not mutually exclusive. 

First, a few States described the lEP review process and other mechanisms within that 
process (e.g., a team monitor) as the primary means of evaluation. The implication was that 
effectiveness, as determined by the individual student’s progress on the lEP, was evaluated by 
the members of the student’s team using the annual review process. In contrast, a number of 
States described the two-tiered approach, discussed earlier, to evaluate the effectiveness of the 
lEP. The most common process generally called for the LEAs to "evaluate" the annual review 
process using methods described in the local annual plans. The results of this evaluation were 
then reported in the LEAs’ annual reports to the SEA. In turn, the SEA "evaluated" the lEP by 
reviewing the LEA plans and reports and by on-site monitoring of the lEP process. Few States 
delineated how the SEA planned to evaluate the effectiveness of the lEP using the data supplied 
by the LEAs. However, some State documents, such as those from Iowa and Washington, 
described rather extensive LEA guidelines for their evaluation of the lEP. 

A third approach, described by only a few States, was to evaluate the lEP at the SEA 
level, independent of evaluations at the local level. The most common method of studying the 
lEP process in these States was to collect a sample of lEPs from districts and analyze them on 
the basis of some criteria. These criteria included the lEP components delineated in 34 C.F.R 
§300.346, as well as the number of goals/objectives attained, parent satisfaction, and participation 
by vocational education and physical education among others. The District of Columbia 
proposed such a study in its State Plan, and South Carolina and Arkansas reported the results of 
similar types of studies in their 89-90 Performance Reports. 

A somewhat different approach to evaluating the lEP is being piloted by Illinois during 
the 1991-92 school year. In this approach, there are three required components of the lEP 
evaluation to be conducted at the local level: (a) evaluate and report findings using the current 
State designated question (i.e.. How often do teachers and related service providers review 
student’s lEPs and for what purposes?), (b) evaluate and report findings using a locally 
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determined question, and (c) using the LEA exit report data for 1991-92, analyze the variations 
in the reasons for exiting of students with different disabilities and report the findings. The LEAs 
were given an "Evaluation Activity Guidebook" to assist in these efforts. The source documents 
for this study did not reveal how the Illinois SEA would compile and analyze these data. 

The lEP plays a central role in planning and documenting services and in ensuring due 
process rights of students with disabilities. It could provide the ideal tool for identifying 
individual student performance outcomes and assessing student progress. In addition, the 
assessment of student outcomes could be used as input into an evaluation of the team process that 
produced the content of the lEP. 

The Development of Outcomes-based Evaluation^ 

Some State departments of special education have begun to investigate ways of collecting 
data on student performance outcomes for the purpose of evaluating the effectiveness of special 
education. At least four States (i.e., Indiana, Michigan, Oklahoma, South Carolina) have 
produced comprehensive technical assistance or guidance documents that encourage LEAs to use 
student outcomes in their evaluation activities. Thus, the use of outcome measures by the LEAs 
in these States is voluntary. In contrast, Connecticut, Delaware, Kentucky, and Maryland are in 
various stages of implementing statewide requirements for collecting student outcome information 
as a basis for LEA and/or SEA evaluation. 

Three States, Vermont, Utah, and West Virginia, reported a process in place for collecting 
statewide information on student outcomes for evaluation purposes. Vermont, for example, is 
developing a statewide educational assessment system for all children. Within this system, the 
performance of special education students will be measured against statewide quality indicators. 
In addition, over half the districts in Vermont participate in the Post School Indicators Project 
which uses student follow-up data to suggest areas for program improvement. 

Utah has developed the Special Education Evaluation System (SEES) to assess the 
operation and effectiveness of special education programs in five areas: (a) student outcomes, (b) 
effective instruction, (c) student records (including lEPs), (d) service coordination, and (e) 
community involvement. The CSPD section of the LEA application in Utah is structured around 
the five areas of SEES. SEES data is collected locally and then organized in a standard format 
for analysis at both the local and State levels. 

The West Virginia Special Education Evaluation Review System (SEERS) has as one of 
its guiding principles, "The focus of the system is upon outcomes, specifically student progress 
and student outcomes." Using this and the other six principles of the SEERS system, LEAs must 



* In this section of the report the reference to States’ efforts to collect student outcome data is in addition to 
the Federal data reporting requirements. 
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involve major stakeholders in the planning and implementation of the evaluation and focus on 
at least one of the established SEERS Program Evaluation Goal Statements. These Goal 
Statements represent major aspects of special education, including eligibility (e.g., decisions 
regarding the student’s eligibility are based upon established criteria), placement, assessment, the 
lEP (e.g., the student’s lEP serves as an effective communication tool), parent involvement, and 
transition (e.g., after program completion, students participate fully in society). SEERS also 
includes a required set of components (i.e., an evaluation plan) for each local evaluation. The 
West Virginia Department of Education annually selects a small number of evaluation questiOiis 
for the purpose of determining the State’s progress in meeting one of these goal statements. 



Summary and Discussion 



This study describes the current status of States’ efforts to evaluate special education 
programs and services. Some States have established a statewide system for collecting, 
analyzing, and reporting evaluation data to interested stakeholders. Several more States are in 
the planning or piloting stage of implementing such a system. Moreover, nearly one third of the 
States support independent evaluation projects to assess the usefulness of curricula, programs, and 
processes implemented to improve the delivery of special education to children and youth with 
disabilities. 

These significant evaluation efforts notwithstanding, the results of this study also indicate 
that the majority of States continue to use compliance monitoring procedures as a program 
evaluation tool to a significant degree, and findings from monitoring activities, presumably, as 
evidence of the effectiveness of special education programs and services. In some States, there 
are clear implications that corrective action following compliance monitoring is equated with 
implementing program improvements. If what States describe in their State Plans and 
Performance Reports accurately reflects their annual program effectiveness evaluation procedures, 
it seems as though program evaluation and compliance monitoring frequently are viewed as the 
same activity. 

There appear to be several reasons why States have not clearly differentiated these 
activities. First of all, the Federal requirements to "evaluate the effectiveness" of special 
education programs, as expressed in both statute and regulation, provide no elaboration on how 
the evaluation process is to be conducted, what data is to be collected, or how it is to be analyzed 
or reported. There also has been very little guidance with respect to the content of the "Annual 
Evaluation" section of the State Plan. 

Second, it would seem natural for many States to want to use, for as many purposes as 
possible, the voluminous amounts of data that must be collected during compliance monitoring 
and fiscal audits, the child count, and other Federal data reporting requirements. The collection 
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of these data require a great deal of effort on the part of both the LEA and the SEA. 
Consequently, using it for more than one purpose is cost effective. 

A third possible factor influencing the apparent emphasis on "monitoring as evaluation" 
is that expertise in program evaluation may not be readily available in many SEAs or LEAs. A 
certain amount of skill and knowledge are required to construct an effective evaluation plan at 
any level within a system. At the State level, it clearly takes more than just knowledge of 
evaluation methods; it takes the equipment, personnel, and statistical expertise to do meta- 
analyses of data on multiple variables supplied from multiple sources. 

Regardless of the reasons behind equating program evaluation with compliance 
monitoring, it is important to remember that monitoring data may contribute information to the 
evaluation process, but by itself, it does not indicate the quality or effectiveness of special 
education services. As DeStefano (1990) stated, "Compliance monitoring simply assesses the 
extent to which a program adheres to a set of standards. It does not in and of itself give a 
comprehensive picture of the program or evaluate program effectiveness" (p. 230). 



The Influence of Student Outcomes on SEA Evaluation 

With the growing emphasis on accountability, the measurement of goals and establishment 
of high performance standards in an era of educational reform, it is not surprising that several 
States are looking at student performance outcomes as a measure of program effectiveness. 
Determining effectiveness is the function of summative evaluation. Patton (1986) tells us that 
"summative evaluation questions are these: Has the program been effective? Should it be 
continued? Is it worthwhile? Did the program bring about (cause) the desired outcomes?" (p. 
66). The answers to these questions are often useful in determining whether or not to continue 
supporting a particular program or project. They do not, unfortunately, provide information on 
how to improve the program or project. To accomplish this, a formative evaluation needs to be 
conducted using a different set of questions and often different data. Questions usually posed 
in a formative evaluation are: "What are the strengths and weaknesses of the program? How can 
the program be improved? What’s working well and what isn’t working so well? What are the 
reactions of clients, staff, and others to the program?" (Patton, 1986, p.66). 

During the course of the OSEP Annual Leadership Conference held in April, 1992, several 
State Directors of Special Education were heard to decry the overall inability to use existing data 
for program improvement. The problem is, of course, that a large amount of the data States 
collect for compliance monitoring and Federal reporting purposes cannot answer formative 
evaluation questions. There must be a conscious effort to collect data that address both program 
improvement and program effectiveness evaluation issues. Also apparent, is the critical need to 
define more precisely a formative and summative evaluation system for special education. 
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Recommendations 



State and local implementation of P.L. 94-142 began in the 1970s with an emphasis on 
establishing fundamental policy and procedural structures necessary to assure access to public 
education for children with disabilities. Compliance monitoring was a necessary force in securing 
the newly granted procedural safeguards as well as the State and local policies to uphold them. 
Since that time, implementation has progressed toward its current focus on improving the quality 
and results of the educational experience for children and youth served under the Act. In order 
to support the present focus in implementation, there should be a renewed emphasis on program 
evaluation as a necessary tool for determining the effectiveness of special education programs 
and services and making informed decisions concerning program improvement. This is not to say 
that compliance monitoring should be discontinued. However, consideration might be given to 
promoting a balance between the human and fiscal resources necessary for conducting 
compliance monitoring and program evaluation at all levels of special education administration 
based upon their relative contribution to improving outcomes for children and youth with 
disabilities. 

A related recommendation is to increase technical assistance in the area of program 
evaluation at the SEA and LEA levels. Expanding the focus of monitoring activities to include 
program evaluation, as the first recommendation suggests, will not necessarily change the amount 
of effort required for data collection. It has the potential, however, to make the data much more 
useful as long as there is a certain level of expertise available to interpret, report, and make the 
improvements the data suggest. For example, the documents reviewed for this study indicated 
that several States adopted portions of the MidSouth RRC evaluation model following technical 
assistance from the RRC staff. The use of this model among several States within the MidSouth 
region suggests that SEAs need and can benefit from technical assistance in program evaluation. 
Again, if the first recommendation is Loilowed, perhaps OSEP’s Division of Assistance to States 
could provide technical assistance in program evaluation and analyze the results of evaluation 
efforts nationwide. 

The present language of the evaluation provisions of the Individuals with Disabilities 
Education Act, the General Education Provisions Act, and many supporting State statutes and/or 
regulations require evaluation for program effectiveness. The results from this study suggest that 
program evaluation should be viewed broadly if it is to be of value in answering questions of 
interest to all stakeholders^. Some of these questions undoubtedly will deal with program 
effectiveness, but other questions of equal importance to individuals who implement special 
education programs will address how these programs can be improved. Thus, a final 
recommendation stemming from this study is that State and Federal laws and regulation should 
be phrased so as to avoid restricting, or otherwise limiting, the types of program evaluations that 



^ One reviewer stressed that "all stakeholders" includes the taxpayers within a given communiiy. Therefore, 
evaluation efforts must be directed toward an open discussion of who, what, where, when, and how much. 
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can be conducted by SEAs and LEAs. As the implementation of the IDEA continues to evolve 
in the coming years, the questions of critical importance to improving program quality, as well 
as determining program effectiveness, will need to reflect these changes. Maintaining the 
flexibility to address the program evaluation needs of the future will be crucial to continued 
progress in implementing a free and appropriate education for children and youth with 
disabilities. 
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